Mining Adverse Drug Reactions from Unstructured Mediums at Scale

Adverse drug reactions/events (ADR/ADE) have a major impact on patient health and care costs. While most ADR’s are not reported via formal channels, they often documented in variety of unstructured conversations such as social media posts or customer support call transcripts. In this paper, we propose natural language processing (NLP) solution that detects free-text conversations, which improves previous work three ways. First, new Named Entity Recognition (NER) model obtains state-of-the-art accuracy for ADR Drug entity extraction the ADE, CADEC, SMM4H benchmark datasets (91.75, 78.76, 83.41% F1 scores respectively). Second, two Relation Extraction (RE) models introduced—one based BioBERT while other utilizing crafted features over Fully Connected Neural Network (FCNN)—perform par with existing models, outperform them when trained supplementary clinician-annotated RE dataset. Third, text classification model, CADEC dataset (86.69% score). The complete is implemented unified NLP pipeline production-grade library built top Apache Spark, making it natively scalable millions records commodity clusters.